This chapter discusses monitoring a SpinServer system by using event monitoring and notification, which are provided by the event command node. It also discusses monitoring the commands that are issued on a SpinServer cluster.
See the SpinFS 2.5 Messages and Troubleshooting Guide for a complete list of system messages and troubleshooting steps.
SpinServer events provide information about all operations that occur on a SpinServer system. The following sections discuss the ways in which SpinServer events are organized.
The following monitoring services and their features are supported:
Syslog Messages: The Syslog service receives event messages from the SpinServer kernel and user applications and directs them to a log file.
Alarms: Sensors are defined for various system elements. The values for these sensors are checked periodically. If a value exceeds a preset threshold, a warning is generated.
Notifications: You can configure user subscriptions for receiving event messages. Subscriptions define the messages in which users are interested and the means by which they are notified. You can configure the frequency with which users are notified of an event.
Logging: A log store logs all event messages. The log messages can be displayed or monitored live. You can also use the SpinLogReader utility to view the logs.
All events include the following information:
Module: The source of the event, such as Syslog, RAID, SpinAccess, etc. See the SpinFS 2.5 Messages and Troubleshooting Guidefor a listing of all modules.
Name: The name of the event.
Severity: The level of severity of the event.
Events can be filtered by IDs, modules, and severities. Filters include the following:
Particular Event Filter: The Particular Event Filter filters messages by message ID
Group Event Filter: The Group Event Filter filters messages by modules and severity
The levels of severity are defined as follows:
EMERGENCY: The SpinServer system is unusable.
ALERT: Action must be taken immediately.
CRITICAL: A critical condition has occurred.
ERROR: An error conditions has occurred.
WARNING: A warning condition has occurred.
NOTICE: A normal but significant condition has occurred.
INFORMATIONAL: Information is generated.
DEBUG: A debug-level message is generated.
The SpinServer audit log lists all of the commands that are issued on a SpinServer cluster. This can be useful for troubleshooting (for instance, whether a command was run incorrectly) and for tracking down possible security problems.
Routine management tasks for monitoring the operational performance of a SpinServer system's hardware and software functions include the following:
Displaying the current state of system alarms
Creating and modifying event filters
Configuring event-notification handlers and subscriptions
Monitoring syslog messages
The event alarm show command displays the current states of alarm variables. Used without the parameter, the following information is displayed: index, instance, variable, status, and current value. Use the -index parameter to display information about a specific alarm. Possible status values are as follows:
OK (The variable is in the normal range.)
Alarm (The variable is in a state that indicates a problem.)
---- (The alarm sensor is not sensing.)
The following example displays the current states of all alarm variables.
spinserver::> event alarm show
Index Module Instance Variable Status Current
Value
0 alarm CPU 1 Fan sensorValue OK 4166
1 alarm CPU 2 Fan sensorValue Alarm 2647
2 alarm Chassis Intake Fan 1 sensorValue Alarm 0
3 alarm Chassis Intake Fan 2 sensorValue Alarm 0
4 alarm Chassis Intake Fan 3 sensorValue Alarm 0
5 alarm Chassis Intake Fan 4 sensorValue Alarm 0
6 alarm Chassis Exhaust Fan 1 sensorValue Alarm 0
7 alarm Motherboard Temperature 1 sensorValue OK 31
8 alarm Motherboard Temperature 1 sensorValue Alarm 31
9 alarm CPU 1 Temperature sensorValue Alarm 16
10 alarm CPU 1 Temperature sensorValue OK 16
11 alarm Motherboard Temperature 2 sensorValue OK 34
12 alarm Motherboard Temperature 2 sensorValue Alarm 34
13 alarm CPU 2 Temperature sensorValue Alarm 164
14 alarm CPU 2 Temperature sensorValue Alarm 164
15 alarm Power supply monitor failed Alarm false
16 alarm CPU Core Voltage sensorValue Alarm 0.0
17 alarm 3.3 Volt Supply sensorValue Alarm 3.268
18 alarm 3.3 Volt Supply sensorValue Alarm 3.268
19 alarm 5.0 Volt Supply sensorValue Alarm 4.966
20 alarm 5.0 Volt Supply sensorValue Alarm 4.966
21 alarm 12.0 Volt Supply sensorValue Alarm 11.8125
22 alarm 12.0 Volt Supply sensorValue Alarm 11.8125
Use the event spinevents commands to create a filter for a particular event and to display configuration information for Spin events.
Spin events include notifications of hardware malfunctions such as low backup battery power, mismatched RAID controllers' firmware, failed RAID set initialization, a disabled UPS, and a failed fan. Spin events also include notifications of incorrect configuration information such as NetBIOS name, server name, password, or virtual server masks that are wrong or do not meet naming conventions.
The event spinevents createfilter command specifies an event ID and filter name for a new Spin event filter. The filter is added to the filter table. The syntax of the command follows:
event spinevents createfilter -msgid event_ID -filtername filter_name
The following example creates a filter named raid_criticals with event ID 82:
spinserver::> event spinevents createfilter -msgid 82 -filtername raid_criticals
To add an event to an existing Spin event filter, use the event spinevents addotfilter command. The command has the following syntax:
event spinevents addtofilter -msgid event_ID -filtername filter_name
The following example adds Spin event 655480 to a filter named raid_criticals:
spinserver::> event spinevents addtofilter -msgid 655480 -filtername raid_criticals
After setting up Spin event filters, you can test them by using the event spinevents sendtestevent command to send test events. If you issue the command with the ID of a single event, the command sends a single message for that event. If you issue the command with the ID of an event group, the command sends a message for each event in the group. You can obtain event IDs by using the event spinevents show and event spinevents show-group commands, as described in the following section.
The command has the following syntax:
event spinevents sendtestevent -msgid event_ID
The following example sends out test messages for each event in event ID 82:
spinserver::> event spinevents sendtestevent -msgid 82
The event spinevents show command displays the following information for Spin events: message ID, module name, event name, event severity, and the number of events for event groups. To display information for a particular event, specify the -msgid parameter. The syntax of the command follows:
event spinevents show [-msgid event_ID]
The following example displays information about Spin event 82:
spinserver::> event spinevents show -msgid 82 Message ID 82 Module raid Name critical_events Severity critical # Events 25
To display all of the Spin events that make up an event group, use the event spinevents show-group command. The command has the following syntax:
event spinevents show-group[-msgid event_ID]
The following example displays the events that make up Spin event 82:
spinserver::> event spinevents show-group -msgid 82
Message Module Name Severity
ID
655372 raid DISK DEAD critical
655385 raid DISK SOFT ID critical
655395 raid LOOPID CONFLICT critical
655401 raid RAIDSET CHECK DISK FAILED critical
655402 raid RAIDSET OFFLINE critical
655409 raid RAIDSET BADBLOCK critical
655420 raid UPS FAILED critical
655422 raid FAN FAILED critical
655425 raid POWER FAILED critical
655428 raid ENCHEAT BAD critical
655434 raid ENC ACCESS OFFLINE critical
655435 raid ENC SOFTADDR OCCURRED critical
655440 raid ENCLOSURE SHUTDOWN critical
655442 raid ENCLCOLD FAIL critical
655452 raid INSTALLATION ABORT critical
655453 raid CTL FIRMWARE MISMATCH critical
655455 raid WARM BOOT ERR critical
655479 raid CTL MIRROR TABLE ERR critical
655480 raid CTL STOP EJECTED critical
655488 raid CHN SCSI BUS DEAD critical
655490 raid CHN FIBRE DEAD critical
655506 raid CTL WRONG FIRMWARE critical
655510 raid CTL RESET FAILED critical
655511 raid CTL RESET ATTEMPT FAILED critical
655512 raid RS NEW RESET FAILED critical
The event log show command displays the log records for a SpinServer. The following information is displayed: event ID, the time it occurred, the severity of the event, the module, last time the event occurred, and a descriptive message about the event. To view the log records for a specific event on a SpinServer, use the -eventid parameter. The syntax of the command follows:
event log show [-eventid event_ID]
The following example displays information about the event that has the ID 123.
spinserver::> event log show -eventid 123
Event filters enable you to specify the warning messages that you want to receive and the information that you want to be sent to a remote SpinServer. Use the event filters commands to create, display, and modify event filters.
Use the event filters new command to create a new event filter by designating the name and message IDs for the filter.
Note: To create a new filter for a particular event, use the event spinevents createfilter command.
The command has the following syntax:
event filters new -name filter_name -msgids message_IDs
To obtain a full list of message IDs, use the event spinevents show command.
The following example creates a new filter named sev1 that initially includes event ID 82:
spinserver::> event filters new -name sev1 -msgids 82
After you have created a filter, you can add events to it by using the event filters addgrpevents command. The command has the following syntax:
event filters addgrpevents -name filter_name -modules [list_of_modules] -severity {debug | informational | notice | warning | error | critical | alert | emergency}
The following examples add alert and emergency events from the dblade and raid modules to the filter named sev1.
spinserver::> event filters addgrpevents -name sev1 -modules [dblade,raid] -severity emergency spinserver::> event filters addgrpevents -name sev1 -modules [dblade,raid] -severity alert
The event filters modify command modifies the message IDs for a specified filter. Whereas the event filters addgrpevents enables you to add messages to a filter incrementally, the event filters modify command can add messages to the message list or remove messages from the list, but the complete message list must be specified in the command. For instance, if you have a message list that contains messages 34, 42, and 58, you can use the event filters addgrpevents command to add message 66. If you then want to remove message 42 from the list, you can use the event filters modify command to specify a message list of 34, 58, and 66.
The command has the following syntax:
event filters modify -name filter_name -msgids message_IDs
The following example modifies the sev1 filter so that it includes messages 34, 42, 58, 66, 74, 82, and 90.
spinserver::> event filters modify -name sev1 -msgids 34,42,58,66,74,82,90
The event filters show command displays the filter name and message IDs for one or more event filters. To display the configuration information for a specific event filter, specify the -name parameter. The syntax of the command follows:
event filters show [-name event]
The following example displays information about a filter named sev1:
spinserver::> event filters show -name sev1 Name Criticals Spin Message IDs 34, 42, 58, 66, 74, 82, 90, 98, 106, 114, 122, 130, 138, 146, 154, 162, 170, 178, 186, 194, 202, 218, 234, 242, 250, 258, 266, 274, 282, 290, 306, 322, 330
The event filters showeventnames command displays the filter name, message names, and message IDs for one or more event filters. The command has the following syntax:
event filters showeventnames [-name event]
The following example displays event-name information about a filter named sev1:
spinserver::> event filters showeventnames -name sev1 Action "Show Message Names" returned with the following status: dblade:critical_events(34) bip:critical_events(42) dimm:critical_events(58) vldb:critical_events(66) cli:critical_events(74) raid:critical_events(82) alarm:critical_events(90) streamdaemons:critical_events(98) software:critical_events(106) vfs:critical_events(114) spinaccess:critical_events(122) audit:critical_events(130) logmsg:critical_events(138) syslog:critical_events(146) kernellog:critical_events(154) jobcontrol:critical_events(162) rootfilesystem:critical_events(170) bootimage:critical_events(178) fcddriver:critical_events(186) hamgr:critical_events(194) ivs:critical_events(202) dbladeincrcopy:critical_events(218) spinfsClient:critical_events(234) rdberrormsg:critical_events(242) vldbClient:critical_events(250) vodClient:critical_events(258) net:critical_events(266) lp:critical_events(274) spiff:critical_events(282) ndmp:critical_events(290) rpc:critical_events(306) antivirus:critical_events(322) backup:critical_events(330)
The event filters delete command deletes an event filter. The command has the following syntax:
event filters delete -name filter_name
The following example deletes the sev1 filter.
spinserver::> event filters delete -name sev1
Use the event notification commands to manage event handlers, to view the history of transmitted notifications, and to register events to be handled.
There are three types of event handlers: email, snmptrap, and user. To maintain the life cycles of event handlers, you can create, display, modify, and delete handlers.
The event notification handlers new command creates a handler of event notifications. For email, the mail server address, followed by a list of email addresses, must be used as the argument of the -parameters parameter. The command has the following syntax:
event notification handlers new -name name -type {email | snmptrap | user} -parameters parameters
The following example creates an event-notification handler named Server1 of type email. The handler sends event notifications to the email addresses pat@acmebricks.com and chris@acmebricks.com on the mail server pop1.acmebricks.com.
spinserver::> event notification handlers new -name Server1 -type email -parameters pop1.acmebricks.com, pat@acmebricks.com, chris@acmebricks.com
The event notification handlers show command displays configuration information for a list of handlers of event notifications. The following information about event handlers is displayed: name of handler, type of handler (email, snmptrap, or user), and the parameters for the handler. To view the configuration information for a specific event handler, use the -name parameter. The command has the following syntax:
event notification handlers show [-name name]
The following example displays configuration information for an event-notification handler named Server1.
spinserver::> event notification handlers show -name Server1
The event notification handlers modify command modifies the configuration of a handler of event notifications. For email, the mail server address, followed by a list of email addresses, must be used as the argument of the -parameters parameter. The command has the following syntax:
event notification handlers modify -name name -parameters parameters
The following example modifies an event-notification handler named Server1 so that it sends email notification to the address pat@acmebricks.com on the mail server named pop1.acmebricks.com.
spinserver::> event notification handlers modify -name Server1 -parameters pop1.acmebricks.com, pat@acmebricks.com
The event notification handlers test command tests an event-notification handler by sending a test message through the handler. The command has the following syntax:
event notification handlers test -name handler_name
The following example tests an event-notification handler named Server1:
spinserver::> event notification handlers test -name Server1
The event notification handlers delete command deletes a handler of event notifications. The command has the following syntax:
event notification handlers delete -name name
The following example deletes an event-notification handler named Server1.
spinserver::> event notification handlers delete -name Server1
The event notification history show command displays the history of event notifications for a specific host. The following information is displayed:
Host name
Event ID
The last time the notification was transmitted
The severity of the event
The notification message that was transmitted
The handler that transmitted the notification
The last time the event occurred
The command has the following syntax:
event notification history show [-host spinserver -eventid event_ID]
To view the log records for a specific event on a SpinServer, use the -host and -eventid parameters.
The following example displays the history of event notifications for the event 123 on the host named Server10.
spinserver::> event notification history show -host Server10 -eventid 123
To register events to be handled, use the event notification subscriptions commands. To manage subscriptions to event notifications, you can create, display, modify, and delete subscriptions.
The event notification subscriptions new command specifies the filters, handlers, threshold values, and threshold time periods (in minutes) for a new subscription to event notifications. The command has the following syntax:
event notification subscriptions new -filter filter_name -handler handler_name -thresholdval threshold_value -thresholdperiod threshold_period
The following example creates a new events-notification subscription that consists of the filter named all_crits and the handler named email_admin. The threshold value is 1 event and the threshold period is 10 minutes. If an event occurs, and if the time since the last transmission is greater than the threshold value, the notification is sent again.
spinserver::> event notification subscriptions new -filter all_crits -handler email_admin -thresholdval 1 -thresholdperiod 10
The event notification subscriptions show command displays the configuration information for notification subscriptions. The following information is displayed:
Module
Event name
Severity level
To display the configuration information for a particular event-notification subscription, use the -filter and -handler parameters. The command has the following syntax:
event notification subscriptions show [-filter filter_name -handler handler_name]
The following example displays the configuration information for an event-notification subscription with a filter named all_crits and a handler named email_admin:
spinserver::> event notification subscriptions show -filter all_crits -handler email_admin
The event notification subscriptions modify command modifies the filters, handlers, threshold values, and threshold time periods (in minutes) of a subscription to event notifications. The command has the following syntax:
event notification subscriptions modify -filter filter_name -handler handler_name -thresholdval threshold_value -thresholdperiod threshold_period
The following example modifies an event-notification subscription named net_all_crits so that it uses a filter named all_crits and a handler named email_admin. It has a threshold value of 2 events and a threshold period of 5 minutes.
spinserver::> event notification subscriptions modify -name net_all_crits -filter all_crits -handler email_admin -thresholdval 2 -thresholdperiod 5
The event notification subscriptions delete command deletes a subscription to event notifications. The command has the following syntax:
event notification subscriptions delete -filter filter_name -handler handler_name
The following example deletes an event-notification subscription with a filter named all_crits and a handler named email_admin:
spinserver::> event notification subscriptions delete -filter all_crits -handler email_admin
The event syslog commands enable you to configure the Spin event-tracking system to send local SpinServer system events to the syslog daemon running on a remote SpinServer. You can specify the hosts to which the events are sent and filter the events that are to be sent.
The event syslog new command creates a host configuration that determines where the Spin events are sent and the filter that is used. The command has the following syntax:
event syslog new -host spinserver -filter filter_name -facility facility
The following example creates a syslog-event configuration in which a SpinServer named Server10 receives event messages from a filter named raid_warn. The facility is specified as user.
spinserver::> event syslog new -host Server10 -filter raid_warn -facility user
The event syslog show command displays a list of host configurations. To display the configuration information for a particular host, use the -host and -filter parameters. The command has the following syntax:
event syslog show [-host spinserver -filter filter_name]
The following example displays configuration information for a syslog event host named Server10 and a filter named all_crits:
spinserver::> event syslog show -host Server10 -filter all_crits
Use the event syslog modify command to modify a syslog-event configuration. The command has the following syntax:
event syslog modify -host spinserver -filter filter_name -facility facility
The following example modifies a syslog-event configuration such that a SpinServer named Server10 receives event messages from a filter named raid_alerts:
spinserver::> event syslog modify -host Server10 -filter raid_alerts -facility user
Use the event syslog test command to test a syslog-event configuration. The command causes a test message to be routed through the configuration. The command has the following syntax:
event syslog test -host spinserver -filter filter_name
The following example tests a syslog-event configuration on a SpinServer named Server10 with a filter named raid_warn:
spinserver::> event syslog test -host Server10 -filter raid_warn
The event syslog delete command deletes a host configuration for Spin events. The command has the following syntax:
event syslog delete -host spinserver -filter filter_name
The following example deletes a host configuration on a SpinServer named Server10 with a filter named all_crits:
spinserver::> event syslog delete -host Server10 -filter all_crits
The audit log lists all commands that have been issued to a SpinServer cluster. Use the system auditlogs show command to view the audit log. The information displayed includes the user that ran each command, the date and time at which the command was run, and the status of the command (Success or Failure). Use the command with the -logid parameter to view information on a specific command. The syntax of the command follows:
system auditlogs show [-logid log_ID]
The following example displays all audit-log entries:
spinserver::> system auditlogs show
Log SpinServer ID Command Path Date
ID
1 Server1 raid subsystems show 10/7/02 8:45:30 AM EST
2 Server1 spinservers show 10/7/02 8:45:37 AM EST
3 Server1 ports show 10/7/02 8:45:38 AM EST
4 Server1 ports show 10/7/02 8:45:40 AM EST
5 Server1 cluster show 10/7/02 8:45:40 AM EST
6 Server1 storage units show 10/7/02 8:45:43 AM EST
7 Server1 filestorage scheduler task show 10/7/02 8:46:07 AM EST
8 Server1 filestorage scheduler schedule show 10/7/02 8:46:08 AM EST
9 Server1 filestorage scheduler job show 10/7/02 8:46:10 AM EST
-- Type 'q' to quit and any key to continue --
The following example displays information about log entry 150:
spinserver::> system auditlogs show -logid 150 Log ID 150 SpinServer ID Server1 IP Address 10.1.1.150 User root Command Path filestorage scheduler task show Command filestorage scheduler task show Date 10/9/02 8:32:07 AM EST Result Success
Copyright © 2004 Network Appliance, Inc.; copyright © 2003, 2004 Spinnaker Networks, LLC. All rights reserved. Specifications subject to change without notice.